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RATE SHAPING IN PER-FLOW OUTPUT 
QUEUED ROUTING MECHANISMS HAVING 
OUTPUT LINKS SERVICING MULTIPLE 
PHYSICAL LAYERS 

Pursuant to 35 U.S.C.120, the priority of the inventors's 
earlier filed provisional application filed on Jun. 27, 1996 
under Ser. No. 60/020,642 is claimed. 

L FIELD OF THE INVENTION 

This invention relates to packet switched communication 
networks and, more particularly, to traffic shaping for caus- 
ing the time multiplexed packet flows at queuing points 
within such networks or network elements to conform to 
specified traffic descriptors. 



II. CROSS REFERENCES TO RELATED 
APPLICATIONS 

For other concurrent filings on traffic shaping see Attor- 
ney Docket No. D/96271P by Christopher J. Kappler et al, 
entitled "Multiple Rate Sensitive Priority Queues for Reduc- 
ing Relative Data Transport Unit Delay Variations in Time 
Multiplexed Outputs from Output Queued Routing 
Mechanisms," Attorney Docket No. D/96271Q1P by Landis 
C. Rogers et ai., entitled "Rate Shaping in Per-Flow Output 
Queued Routing Mechanisms for Statistical Bit Rate 
Service," Attorney Docket No. D/96271Q2P by Joseph B. 
Lyles et al, entitled "Rate Shaping in Per-Flow Queued 
Routing Mechanisms for Available Bit Rate Service," Attor- 
ney Docket No. D/96271Q3P by Joseph B. Lyles, entitled 
"Rate Shaping in Per-Flow Output Queued Routing Mecha- 
nisms for Available Bit Rate (ABR) Service in Networks 
Having Segmented ABR Control Loops," Attorney Docket 
No. D/96271Q4P by Joseph B. Lyles et al., entitled "Rate 
Shaping in Per-Flow Output Queued Routing Mechanisms 
for Unspecified Bit Rate Service." 

III. BACKGROUND OF THE INVENTION 
A. Traffic Contracts/Definitions 

Most applications that are currently running on packet 
switched communication networks can function acceptably 
with whatever bandwidth they happen to obtain from the 
network because they have "elastic" bandwidth require- 
ments. The service classes that support these applications is 
known as "best efforts" service in the Internet community 
and as "Available Bit Rale" (ABR) in the Broadband ISDN/ 
ATM community. 

There is, however, a growing demand for network ser- 
vices that provide bounded jitter or, in other words, bounded 
packet delay variation (commonly referred to as cell delay 
variation in an ATM context). For example, this type of 
service is required for real time applications, such as circuit 
emulation and video. It is not clear whether and how the 
Internet community will respond to this demand, but the 
Broadband ISDN/ATM community has responded by intro- 
ducing the notion of a user-network negotiated traffic con- 
tract. 

As is known, a user-network ATM contract is defined by 
a traffic descriptor which includes traffic parameters, toler- 
ances and quality of service requirements. A conformance 
definition is specified for each of the relevant traffic param- 
eters. Accordingly, ATM services may make use of these 
traffic parameters and their corresponding conformance 
specifications to support different combinations of Quality 
of Service (QoS) objectives and multiplexing schemes. 

Partially overlapping sets of ATM traffic classes have 
been defined by the Telecommunications Standardization 



Sector of the International Telecommunications Union 
(ITU-T) and the ATM Forum. In some instances, traffic 
classes which have essentially identical attributes have been 
given different names by these two groups, so the following 
name translation table identifies the existing equivalent 
counterparts: 



10 



rrU-T Traffic Class 



ATM Forum Traffic Class 



ABR 

Deterministic Bit Rate (DBR) 

Statistical Bit Rate (SBR) 
(No existing counterpart, but 
understudy) 
35 (No existing counterpart, but 
understudy) 



ABR 

Constant Bit Rate (CBR) 
Variable Bit Rate (VBR) 
Real time Variable Bit Rate (rt-VBR) 

Unspecified Bit Rate (UBR) 



An ATM service contract for a virtual circuit (VC) con- 
nection or a virtual path (VP) connection may include 

20 multiple parameters describing the service rate of the con- 
nection. This includes the Peak Cell Rate (PGR), the Sus- 
tainable Cell Rate (SCR) the Intrinsic Burst Tolerance (IBT), 
and the Minimum Cell Rate (MCR). Not all of these 
parameters are relevant for every connection or every ser- 

25 vice class, but when they are implied or explicitly specified 
elements of the service contract, they must be respected. VC 
connections are the primary focus of the following 
discussion, but it will be understood the VP connections can 
also be so specified. The data transport unit for an ATM 
connection usually is referred to as a "cell." In this 
disclosure, however, the term "packet" is sometimes used to 
refer to the data transport unit because this more general 
terminology is consistent with some of the broader aspects 
of the innovations. 

The Generic Cell Rate Algorithm (GCRA), which is 

^5 specified in ITU-T Recommendation 1.371, is well suited 
for testing a packet or cell flow for conformance with a 
traffic descriptor. To perform such testing, the GCRA 
requires the specification of an emission interval (i.e., the 
reciprocal of a flow rate) and a tolerance, t. In practice, this 

40 tolerance may depend on a variety of factors, including the 
connection, the connection setup parameters, or the class of 
service. As will be seen, the GCRA can be employed as a 
Boolean function, where for a flow of fixed size packets or 
cells on a connection, the GCRA (emission interval, 

45 tolerance) is false if the flow is conforming to a peak rate or 
true if the flow is conforming to a minimum rate. For 
example, a source of cells conforms to a PCR if GCRA 
(1/PCR, TpcR ) is false. Likewise, a connection or flow 
conforms to an MCR if GCRA (1/MCR, x^^id ^ false. As 

50 will be appreciated the "emission interval" is the reciprocal 
of the "cell rate." 

A DBR traffic contract is appropriate for a source which 
establishes a connection in the expectation that a static 
amount of bandwidth will be continuously available to the 

55 connection throughout its lifetime. Thus, the bandwidth the 
network commits to a DBR connection is characterized by 
a PCR value. Further, the cell or packet flow on such a 
connection complies with the traffic contract if it conforms 
to GCRA (1/PCR, Xpcr). On the other hand, an SBR traffic 

60 contract is suitable for an application which has known 
traffic characteristics that allow for an informed selection of 
an SCR and x^bt^ as well as a PCR and XpcR. An SBR or 
rt-SBR flow complies with its traffic contract if the flow not 
only conforms to GCRA (1/PCR, Vc/O. but also to GCRA 

65 (l/SCR,X/flr) 

As previously indicated, an ABR traffic contract is appro- 
priate for applications that can tolerate the dynamic varia- 
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tions in the information transfer rate that result from the use whilst meeting QoS objectives and subsequent 

of unreserved bandwidth. A PCR and an MCR are specified conformance tests at the network or network seg- 

by the source establishing such a connection, and these ment egress. 

parameters may be subject to negotiation with the network, TrafBc shaping may also be used within the customer 

Thus, the bandwidth that is available on an ABR connection 5 equipment or at the source in order to ensure that the cells 

is the sum of the MCR (which can be 0) and a variable cell generated by the source or at the UNI are conforming to the 

rate that results from a sharing of unreserved bandwidth negotiated traffic contract relevant to the ATC that is used 

among ABR connections via a defined allocation policy (i.e., (^^^ Section 5.5)." ITU-T Recommendation I .371, Section 

the bandwidth a source receives above its specified MCR 6.2.5. 

depends not only on the negotiated PCR, but also on lO ^ Scheduling for Real Time and Non-Real Time 

network policy). Feedback from the network enables the Connections/Existing Tools and Techniques 

source application to dynamically adjust the rate it feeds ^ ^V*^""^"' >f "andwidth is not divided "fairly between 

„ 1 . • AT^Ti • A AT^r, r, applications employing best efforts Internet service or 

cells or packets into an ABR connection An ABR flow l^^- ^^desirable phenomena may 

^^n^r f.T/^' X '"J" conforms to Lefelhocz, Lyles, Shenker and Zhang, "Conges- 

GCRA (1/MCR, Tj^cjd, and is always non-comphant if it 35 tion control for Best-Effort Service: Why we need a new 

does not conform to GCRA (1/PCR, t^^. Conformance in paradigm ,"/££•£: Network, January/February 1996, for fur- 

the region between MCR and PCR is dependent on the ABR ther details on mechanisms for best effort/ABR traffic, 

feedback and is thus dynamically determined. Most ATM switches currenUy are implemented with FIFO 

A UBR traffic contract is similar to the ABR contract, queuing. FIFO queuing exhibits pathological behaviors 

except that the UBR contract does not accommodate the 20 when used for ABR traffic (see "On Traffic Phase Effects in 

specification of an MCR and has no dynamic conformance Packet-Switched Gateways", Sally Floyd and Van Jacobson, 

definition. Therefore, a UBR flow complies with its traffic Internetworking: Research and Experience, Vol, 3, pp. 

contract if it conforms to GCRA (1/PCR, Xj^cr)- 115-156 (1992), and "Observations on the Dynamics of a 

B. Traffic Shaping Congestion Control Algorithm: The effects of Two-Way 

ITU-T Recommendation I ,371 addresses the possibility 25 Traffic", Lixia Zhang, Scott Shenker, and David Clark, A CM 

of reshaping traffic at a network element for the purpose of Sigcomm 91 Conference, Sep. 3-6, 1991, Zurich, 

bringing the traffic into conformance with a traffic descriptor Switzerland, pp. 133-148.). FIFO also is unable to protect 

in the following terms: correctly behaving users against misbehaving users (it does 

"Traffic shaping is a mechanism that alters the traffic not provide isolation). As a result of these deficiencies 

characteristics of a stream of cells on a VCC or a VPC 30 non-FIFO queuing mechanisms such as weighted fair queu- 

to achieve a desired modification of those traffic iog (see, for example, A. Demers, S. Keshave, and S. 

characteristics, in order to achieve better network effi- Shenker, "Analysis and Simulation of a Fair Queuing 

ciency whilst meeting the QoS objectives or to ensure Algorithm," Proceedings of ACM SiqComm, pages 1-12, 

conformance at a subsequent interface. Traffic shaping September 1989; and A. K. Parekh "A Generalized Proces- 

must maintain cell sequence integrity on an ATM 35 sor Sharing Approach to Flow Control in Integrated Service 

connection. Shaping modifies traffic characteristics of a Networks," Ph.D. Thesis, Department of Electrical Engi- 

cell flow with the consequence of increasing the mean neering and Computer Science, MIT, 1992.) or approxima- 

cell transfer delay. lions to fair queuing such as round-robin (Ellen L. Hahne, 

Examples of traffic shaping are peak cell rate reduction, "Round-robin Scheduling for Max-Min Fairness in Data 

burst length limiting, reduction of CDV by suitably Networks." IEEE Journal on Selected Areas in 

spacing cells in time and queue service schemes. frnftrs^'^'^tel''^" P^'^^^"^^^^^' September 1991.) 

It is a network operator's choice to determine whether and ^ • 1 u-uu i^ uj -a.i. 

• • e A A 1 Service classes which have inelastic bandwidth require - 

where trainc shaping is performed. As an example, a . f* • . j . i_ * j .1. l .1. 

_ 1 * ™ u * c * «r t. . ments often require that data be transmitted through the 

network operator may choose to perform traffic shaping ^ , , J* j j /■ u ^ a n 1 * 

• *• '.X. ut TTTiA^/xm^ n *^ ^ 45 network with bounded utter (i.e., bounded cell or packet 

in conjunction with suitable UPC/NPC functions. •.•\au u\u u f 

delay vanation). As shown by the above referenced Parekh 

It us an operator s option to perform traffic shaping on p^p^^, weighted fair queuing can be used to provide bounded 

separate or aggregate cell flows. ji^^, .^^j ^-^^ ^^^^^^^ Moreover, Parekh^s results have 

As a consequence, any ATM connection may be subject to recently (Pawan Goyal, Simon S. Lam and Harrick M, Vin, 

traffic shaping. 50 "Determining End-to-End Delay Bounds in Heterogeneous 

The options available to the network operator/service Networks" Proceedings of The 5th International Workshop 

provider are the following: on Network and Operating System Support for Digital A udio 

a. No shaping and Video(NOSSDAV), Durham, N.H., Apr. 18-22, 1995.) 
Dimension the network in order to accommodate any been extended to prove delay bounds for systems using the 

flow of conforming cells at the network ingress 55 closely related mechanisms of Virtual Clock (Lixia Zhang, 

whilst ensuring conformance at the network "Virtual Clock: A New Traffic Control Algorithm for Packet 

egress without any shaping function. Switching Networks," Proceedings of ACM SigComm, 

b. Shaping pages 19-29, August 1990.) and Self -clocked Fair Queuing 
Dimension and operate the network so that any flow (S, J, Golestani, "A Self-Clocked Fair Queuing Scheme for 

of conforming cells at the ingress is conveyed by 60 High Speed Applications," Proceedings oflNFOCOM, pp. 

the network or network segment whilst meeting 636-646, 1994). 

QoS objectives and apply output shaping the traf- Thus, it is known that both elastic (Best effort/ABR) and 

fic in order to meet conformance tests at the inelastic (or real-time) services can benefit from the use of 

egress. fair queuing and related algorithms. 

Shape the traffic at the ingress of the network or 65 1. Weighted Fair Queuing and Virtual Clock 

network segment and allocate resources according Fair queuing and related algorithms (e.g., frame-based 

to the traffic characteristics achieved by shaping, fair queuing, deficit round robin, etc.) operate on sequences 
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of packets or other data transport units (e.g., an ATM cell is 
a packet). For ATM these sequences are identified by either 
the VCI or the VPI, while in the Internet protocol suite the 
identification is on the basis of <IP address, protocol, 
port>triples (IPv4) or flow identifiers (IPv6). In both self- 
clocked weighted fair queuing and virtual clock, packets are 
ordered (sorted) by timestamps (schemes such as round- 
robin provide approximations to ordering of packets by 
timestamps). These timestamps represent the virtual finish- 
ing time (or equivalently the virtual starting time for the 
packet and are computed by taking a starting time value and 
adding an ofiket obtained by multiplying the length of the 
packet by a weight which represents the particular packet 
sequencers share of the bandwidth. 

More particularly, for virtual clock the virtual finishing 
time is computed as: 

vr(f, o)-o 

VT(^;+l)-niax{Arrival(/; /+!), Vr(f, /)}+Length(/ /+l)/RateCO(l) 

where: VT(f, j) is the virtual finishing time associated 
with packet j of flow (virtual circuit) f; 
Arrival(f, j) is the arrival time of packet j of flow f; and 
Length(f, j) is the length of packet j of flow f. 

Self -clocked weighted fair queuing assigns virtual finish- 
ing times according to the formula: 

VT(/; o)=o 

Vr(f, ;+l)=max{SystemVirtualTimc, W(f, /)}+Length(/; * 
weight(0 (2) 

where: System VirlualTime is the virtual time associated 
with the packet being served (being output) at the time 
packet(f, j+1) arrives. 
For ATM the packet length is constant because the cells 
are of fixed size (i.e., 53 bytes long). Consequently, right- 
most term in both Expression (1) and Expression (2) 
becomes a per flow constant. For virtual clock the simplified 
expression is: 

Vr(f, /+l)-inax{ArTival(/; VT(f, ;")}+constam(/) (3) 

For self -clocked weighted fair queuing, on the other hand, 
the simplified expression is; 

VT(^ y+l)-max{SystcinVirtualTiinc, VT(f, »}+coiislaDt (f) (4) 

In other words, an ATM queuing point which implements 
either virmal clock or self-clocked weighted fair queuing 
performs the following steps: 

1) compute the maximum of (a) the current virtual time 
for the VC, and (b) either of i) the arrival time of the 
cell or ii) the system virtual time. 

2) add to the results of step 1 above a per-VC constant 
representing that VC's share of the bandwidth. 

3) service cells (transmit them) in order of increasing 
values of the virtual time stamps assigned by steps 1 
and 2. 

2. Priority 

Giving priority to one trafiBc class over another means that 
if the higher priority traCBc class has cells ready for 
transmission, those cells are always transmitted in prefer- 
ence to the cells of the lower priority traffic class. 

Priority mechanisms can be either preemptive or non- 
preemptive. Ihis terminology comes from the operating 
system Ulerature. A non-preemptive priority mechanism 
assigns a priority to an object (a process in the operating 
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system world, a VC in the ATM world) at a scheduling time, 
and the object then retains this priority until it is served. 
Preemptive priority mechanisms, on the other hand, can 
change the priority of objects while they are waiting to be 
5 served. For example, in a preemptive system one could say 
"schedule this VC with priority 3 but if it is not served within 
200 microseconds then increase its priority up to 2." 

3. Work Conserving and Non-work Conserving Queuing 
Klein rock. Queuing Systems, Vol. 2: Computer 

10 Applications, John Wiley & Sons, N.Y, N.Y 1996, p. 113 
uses the terminology "work conserving" to denote any 
queuing system in which work is neither created nor 
destroyed. In keeping with this terminology a switch which, 
when given queued cells, always transmits cells on the 

IS outgoing link is a "work conserving switch**. Switches 
employing a pure FIFO, weighted fair queuing or virtual 
clock scheduling algorithm are all work conserving. In 
contrast, a non-work conserving switch may choose not to 
send cells, even when ceUs are queued for transmission. As 

20 will be seen, a method of doing this is to program the switch 
to wait until the current time is equal to or greater than the 
timestamp associated with a particular cell before transmit- 
ting that cell. 

Work conserving switches attempt to fully utilize the 

25 transmission link, but do not necessarily remove or prevent 
bursts. In contrast, non-work conserving switches can stra- 
tegically delay cells so as to re-shape trafiic to meet a more 
stringent conformance test (i.e., a GCRA with a smaller x). 
Additionally, a non-work conserving switch in which a 

30 given connection is only allocated a specified amount of 
buffering can perform a policing function (in ITU terms a 
UPC/NPC) by discarding or tagging cells which overflow 
the allotted buffer space. An example of a non-work con- 
serving queuing system is the Stalled Virtual Clock (Sugih 

35 Jamin, "Stalled Virtual Qock" working note. Department of 
Computer Science, UCLA, Mar. 21, 1994), which is an 
adaptation of Lixia Zhang's Virtual Clock algorithm where 
virtual time is not allowed to run faster (it stalls or goes 
non-work conserving) than real-time. Also see, work by 

40 Scott Shenker that is available by FIT at FfRPARCX- 
EROX.com. 

4. Calendar Queues 

A calendar queue is a time ordered list of actions, each of 
which is dequeued and executed when real-time is equal to 

45 or greater than the time associated with the action. Calendar 
queues with bounded time intervals can be represented as a 
linear array which is known as a "lime-wheer' or "time- 
hne." Time-wheels assign events to buckets relative to a 
pointer, where the bucket index is calculated using arith- 

50 metic modulo the wheel size. These data structures are well 
known in the literature as a queuing mechanism. In a 
time-wheel, absolute time is represented as an offset relative 
to the current time ("real time"), and each element in the 
array is a bucket which contains one or more actions 

55 (typicaUy in flnked-list form) which are to be executed at the 
time assigned to the bucket in which they reside. Any of the 
buckets of such a time-wheel can be empty, i.e., have no 
events associated with it. 

For every time -wheel, there are two times of interest: 

60 Xeariiest ^^d \iatesn which corTcspond to the head and tail 
pointers for the active entries in the array; where 
the time of the next entry (e.g., packet or cell) to be serviced, 
and *s the time associated with the latest (most distant 
in time) bucket containing a scheduled event. The difference 

65 between igartiest and \iateat cannot be greater than the length 
of the time-wheel, b, minus 1. lliis can be ensured by 
viewing the time as being kept modulo b, and by then 
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ensuring that no ofiset (the packet length multiplied by either FIG. 1 is a simplified block diagram of an ATM switch in 

the rate or the weight in virtual clock or weighted fair which the present invention may be used to advantage; 

queuing respectively) is greate^^ than b-1 For an ATM link piG. 2 diagrammatically tracks the various shapes an 

runnmgatOC-3speeds(14976mbp^theSONETpayl^ ^^1^ ^^^^^^ ^^^^ ^^^^ traversing the switch 

rate) there are approximately 353208 cells/sec on the hnk. 5 jJjq^q ^ FIG 1* 

Accordingly, if 64 Kbps (voice telephony rates) flows ' ' 

(approximately 174 cells/sec when AAL type 1 is used) are ^IG. 3 is a more detailed block diagram of a reprcsenta- 

the lowest speed connections that need to be supported, then ^^^^ channel on the output or transmit side of the chip shown 

the ratio of the highest supported rate to the lowest rate is ^ F^^- 

2029, which rounds up to 2^^. This ratio is the maximum FIG. 4 schematically illustrates a stalled virtual clock 

offset that will get added during the calculation of virtual calendar queue technique for shaping SBR trafBc; 

times. Therefore atime-wheeloflenglh2030(204«toallow 5 schematically illustrates another stalled virtual 

for roundmg up to a power of two) is sufficient to encode the ^^^^^ ^^j^^^^^ ^^^j^^. . ^^^^ 

vurtual times associated with circuits rangmg in rates from r^j^^ ^ ■ . . 

64 Kbps to fiill OC-3 link rate r\G. 6 schematically illustrates a stalled virtual clock 

The length of a time-wheel array can be decreased by 15 calendar queue technique for shaping ABR traffic having 

permitting an array element to contain more than one time MCR guarantees; 

offset. For example, if the above-described time-wheel is FIG. 7 schematically illustrates another stalled virtual 

reduced to 256 elements from 2048, then each bucket would clock calendar queue technique for shaping ABR traffic 

have eight time oflkets mapped into it. Actions within a having MCR guarantees; 

single bucket that spans multiple ofi&ets may be performed 20 piG. 8 schematically illustrates still another stalled virtual 

out of order, but between buckets actions v^^^^ ^^^^^ ^^^^^^^ queue technique for shaping ABR traffic 

This reduces the amount of memory that needs to be , . . 

allocated to such a time-wheel at the cost of reducing the guarantees; 

precision of the ordering of actions in the calendar queue. FIG. 9 schematically an ABR connection having a source- 

D. Traffic Shaping for Time Multiplexed Flows on Multiple 25 to -destination control loop; 

Output Channels FIG. 10 schematically an ABR connection having seg- 

Preferably, any trafBc shaping that is needed to bring time mented control loop; 

multiplexed packet or ceU flows into conformance with dieir schematically illustrates an output link servicing 

traffic contracts is performed after the completion of aU ^^j^. ,^ . .^^j ' 

switchmg or routmg operations that are required to separate r r ^ / » 

flows for different output channels from each other. This . FIG. 12 schematicafly illustrates a calendar queue time- 

permits the throughput efficiency of the multiplexer to be line that is striped with multiple physical layer significance, 
optimized. 

However, prior output queued ATM switches generally ^* DETAILED DESCRIPTION OF THE 
have employed FIFO (First In— First Out) output buffers. INVENTION 
These buffers are not capable of participating in a controlled 35 ^j^^^ invention is described in some detail herein- 
reshaping of any of the flows that pass through them. ^elow with specific reference to certain embodiments, it is 
Instead, the per- VC time multiplexed flows that are output ,^ understood that there is no intent to limit it to those 
by these buffers essentially are time multiplexed composites embodiments. On the contrary, the intent is to cover all 
of the input flows that are loaded into them. Of course, these u *• j • 1 . <• n- m.- *u 

. ^ J J , J w * *u • modifications, alternatives and equivalents fallmg within the 

output flows are time delayed relative to the input flows 40 , r.. .10,, , 1, 

because of the inherent latency of the buffers. Moreover, the ^"'^ °^ ''^S"^'' ''^ aPPended 

cell delay variation (CDV) of one or more of these output claims. 

flows may be increased if scheduling conflicts occur among A. A Representative Environment 

the data transport limits of the different flows because these Turning now to the drawings, and at this point especially 

conflicts cause so-called "transmit collisions," 45 to FIG. 1, the input and output ports of an ATM switch 21 

As will be appreciated, increased CDV is especiaUy typically are coupled to one or more physical layers via 

troublesome for traffic, such as DBR traffic, which generafly respective Utopia 2 interfaces and to a switch control 

has a relatively tight tolerance. Thus, if each hop between a processor module 22 via a second suitable interface, lliis 

source and a destination includes a simple FIFO output enables the switch 21 to exchange data and control ceUs with 

queue of the foregomg type, it may be necessary to hmit the physical layers that are connected to them, and to also 

number of hops this CDV sensitive traffic is permitted to exchange control cells with the control processor module 22. 

make in order to ensure compliance within its specified keeping with standard practices, the communication 

tolerance. channels are unidirectional, so a pair of channels are 

Accordingly, that there is a need for more efficient and i^^^ bi-directional communications, 

more effective traffic shaping mechanisms and processes for ™_ , .^^ • r.. 

* u J *u * *u * . . az e 55 The switch 21 comprises a switching fabric 24, a fabnc 

ATM switches and other routers that route traffic from , , ^ i ^ *• • -iir r i_- 

u- ^ • 4 ♦ 14- 1 * * f w 1 J control module 26 and a reservation nng 26 for switching 

multiple inputs to muUiple outputs for time multiplexed . . , * i n r ■ * . x/o * . 

, ^ • r f r ^^(^ gjjjj control cells from input queues to per-VC output 

output emission. n • »u * j • *u j * *u 

queues. The cells in these queues are stored in the data path 

IV. SUMMARY OF THE INVENTION in data memory 27, and these input and output queues are 

lliis invention provides rate shaping in per-flow output ^0 managed by a queue control module 28. Typically, the data 

queued routing mechanisms having output links servicing memory 27 is sized to hold up to roughly 12000 cells. 

multiple physical layers. Connection records for the data and control cell flows are 

stored in the control path in control memory 29, together 

V. BRIEF DESCRIPTION OF THE DRAWINGS certain types of control cells which are intercepted by 

Still further objects and advantages of this invention will 65 a rate based engine and traffic multiplexer 31 for routing to 

be evident when the foflowing details and description is read the control processor module 22. Suitably, the control RAM 

in conjunction with the attached drawings, in which: 29 is capable of accommodating up to about 8200 connec- 
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tion records and 64 k cell records. The interaction of the 
control processor module 22 with the switch 21 is beyond 
the scope of the present invention and, therefore, is not 
described herein. However, persons who are familiar with 
ATM switch design will understand that the control proces- 
sor is primarily responsible for performing connection estab- 
lishment and termination, as well as 0AM (Operation and 
Maintenance) functions. 

The data path of the switch 21 is synchronously clocked 
(by means not shown) at a predetermined rate of, say, 40 
MHz. However, in keeping with conventional synchronous 
pipeline design practices, the phase of this clock signal is 
delayed (by means also not shown) by differing amounts at 
different points along the data path to give the data adequate 
time to settle prior to being transferred from one pipelined 
stage to the next. 

In accordance with standard practices, a source wishing to 
communicate with a destination initiates negotiations with 
the ATM network within which the switch 21 resides by 
sending a SETUP message to the network. This message 
identifies the destination and explicitly or impliedly specifies 
all of the relevant traffic parameters for the requested con- 
nection. If the network is prepared to commit to the traffic 
contract which is defined by these traffic parameters (or a 
modified version of the parameters that the source is willing 
to accept), the network routes the SETUP message to the 
destination. Then, if the destination is ready to receive 
message traffic from the source in accordance with the terms 
of the traffic contract, the destination returns a CONNECT 
message to the source. This CONNECT message confirms 
that a connection has been established on a specified virtual 
circuit (VC) within a specified virtual path (VP) for a cell 
flow that conforms to the traffic contract. See ITU-T Rec- 
ommendation 0.2391 and ATM Forum UNI 4.0 Specifica- 
tion. "Permanent" virtual connections can be established by 
provisioning, without invoking these signaling protocols. 

Data cells begin to flow after a connection is established. 
As shown in FIG. 2, the form of the cells change as they pass 
through the switch 21 because of the operations, the switch 
performs. Cells may be replicated within the switch 21 for 
multicasting, but the following discussion will be limited to 
unicast operations to avoid unnecessary complexity. 

As indicated in FIG. 2 at 41, each inbound cell that the 
switch 21 receives has a header containing a VP index and 
a VC index. '^ITiese indices combine to define a unique 
address for one hop of the connection. A connection may be 
composed of multiple hops, so the VP and VC indices for the 
next hoop are written into the header of the cell as it passes 
through the switch 21 as indicated in FIG. 2 at 42. 

The switch 21 employs the VP and VC indices of the 
inbound cell (FIG. 2 at 41) to compute the address at which 
the connection record for the associated flow resides within 
the control RAM 29, Typically, this connection record 
includes a bit vector for identifying the output port (i.e., the 
switch -level "destination") at which the flow exits the switch 
21, a priority index for identifying the relative priority of the 
flow on a granular priority scale, and a circuit index 
("Circuit Index") which uniquely identifies the flow inter- 
nally of the switch 21. As shown in FIG. 2 at 43, these 
connection parameters are written into the cell header. Then, 
the cell is written into the data RAM memory 27, while a 
pointer to the cell is linked into an appropriate one of a 
plurality of FIFO input queues, where the selection of the 
queue Ls ba.sed on the priority of the related flow. 

llie relative priorities of the head of queue cells within 
these input queues are examined during each cell time, and 
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the head of queue cell having the highest priority is selected 
for arbitration during the next arbitration session. 
Furthermore, the priority of any lower priority head of queue 
cell (i.e., any non -selected head of queue cell) is incremen- 

5 taUy increased (by means not shown), thereby increasing the 
probability of that cell being selected for arbitration during 
the next arbitration session. Therefore, even though the 
higher priority input queues have greater throughput per unit 
time than the lower priority queues, the lower priority 

10 queues have bounded delay because the priority of their 
head of queue cells increases as a function of time. 

Each arbitration cycle requires one cell time of the switch 
21, so the routing information for the cells that are selected 
for arbitration is fed into the reservation ring 26 one cell time 

15 prior to the release into the switching fabric 24 of the pay 
loads of the cell or cells that win the arbitration. In other 
words, as shown in FIG. 2 at 44, the cells that are received 
by the reservation ring 26 and switching fabric 24 are 
composed of the headers of the ceUs for the next arbitration 

20 cycle (i.e., the "current cells") followed by the bodies or 
payloads of the cells successfully arbitrated during the 
previous arbitration cycle (i.e., the "previous cells"). Thus, 
when the cell bodies reach the fabric 25, the fabric is already 
configured by the fabric control 25 to route those cells to 

25 their respective output port destination. For additional infor- 
mation on the reservation ring 26 and the arbitration it 
performs, see Cisneros, A,, "Large Packet Switch and Con- 
tention Resolution Device," Proceedings of the XIII Inter- 
national Switching Symposium ^ 1990, paper 14, Vol. Ill, pp. 

30 77-83 and Lyles U.S. Pat. No. 5,519,698, which issued May 
21, 1996. 

In the illustrated embodiment, cells are decomposed into 
four bit wide nibbles for arbitration and routing. Thereafter 
(with the exception of "idle cells" that may be provided for 
testing the switching processes) the ceUs are reassembled 
and queued in the data path, the control path, or both (a) for 
time scheduled transfer to the appropriate output ports of the 
switch 21 and/or (b) for transfer to the control processor 
module 22. ITie time scheduled transfer of cells to the output 
ports of the switch 21 is central to this invention so that 
subject is discussed in further detail hereinbelow. On the 
other hand, the decomposition and reassembly of the cells, 
the testing processes, and the interaction of the control 
processor 22 with RM (Resource Management) and OAM 
(Operation and Maintenance) cells are incidental topics 
which need not be considered in depth. 

Referring to FIG. 3, it will be understood that the switch 
21 fans out on the output or transmit side of the switching 
fabric 24. Thus, while one output channel of the switch 21 
is shown, it wiU be understood that this channel is generally 
representative of the other channels. 

As shown, there suitably is a fill cells module 51 for 
accepting cell bodies and their associated circuit indices 

55 from the switching fabric 24. The "effective cell time" on the 
output side of the switching fabric 24 is determined by the 
ratio of the nominal cell time to the "k** speed-up factor. 
Thus, for example, if the nominal cell time is 113 clock 
cycles/cell, the effective ceU time on the output side of the 

50 switching fabric 24 is 56.5 cycles/cell if k=2. 

When a valid cell is received, the fill ceU module 51 
typically uses cell structures from a hnked and numbered 
free list 52 of such data structures for writing the cell into the 
data memory 27. To that end, the fill cell module 51 suitably 

65 includes a fetch state machine 53 for fetching cell structures 
form the top of the free list 52 on demand, lliis enables the 
fill cell module 51 to insert the circuit index for the cell and 
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a pointer to the location of the cell in data memory 27 into per-VC queues. Thus, the calendar queue 63 may be imple- 
an" arrival" message that it sends to notify a cell flow control mented by employing one or more time bounded time- 
unit 55 of the cell's arrival. The circuit index enables the wheels or "time lines," 66. Please see Section ni.C.4 above, 
flow control unit 55 to ascertain the VC or flow to which the The time span of these time -wheels must be at least as long 
cell belongs from the connection record in the control 5 as the period of the lowest frequency flows that the system 
memory 29. This, in turn, enables the cell flow control unit is designed to support to prevent time wrap induced ambi- 
55 to check the traffic shaping status of the flow. An guities and preferably is twice as long so relative times can 
OAM/RM recognizer 27 advantageously is provided to be compared using two's complement computations, 
enable the flow control unit 55 to identify these control cells B. Shaping Flows of Fixed Bit Length Data Transport Limits 
and to determine whether they are to be queued in the data 10 to Specified Peak Flow Rates 

path, the control path, or both. Referring to FIG. 4, it wiU be evident that stalled virtual 

The memory pointers for cells of traffic contract compli- clock transmission control is well suited for shaping time 

ant flows are queued in per-VC queues in response to multiplexed flows of fixed bit length data transport units, 

"addCell" messages which the cell flow control unit 55 such as ATM cells, from an output queued routing mecha- 

sends to a queue control unit 58. Each addCell message 15 nism to specified peak data unit flow rates, such as PCRs for 

identifies the cell to which it pertains and the circuit index DBR/CBR ATM service. As previously described, the data 

for the associated flow or VC. Moreover, the addCell transport units of the flows that are routed to a given output 

message also indicates whether the ceU is to be queued in the Port are queued, after being routed, in per-flow queues. The 

data path, the control path, or both. When the ceU has been data transport units at the heads of these queues then are 

appropriately queued, the queue control unit 58 returns an 20 admitted by an admission controUer 61 (to the exclusion of 

"added" message to the cell flow control unit 55 for noti- all other transport units) for scheduling on a lime fine 

fying the flow control unit 55 that the newly queued cell calendar queue 63 by a scheduler 62. The scheduler 62, m 

needs to be taken into account during future rate shaping turn, performs per-flow virtual clock computations on these 

computations on the VC to which it belongs. head of queue transport units to schedule them for release 

A J 4 1 *u * 1 V CO ™ * *u 25 from the calendar queue 63 in accordance with their respec- 

Advantageously, the queue control unit 58 monitors the ^^^^i^ inv^ vaiv^u^oi mu uv ^ 

1 *u f*L \rr^ uu *t tu A *u * 1 tive theoretical finish times, VT(f, 1+1), or their respective 

length of the per- VC queues with respect to the depth control , , , . ^, ' o / th ^ ^ i_ 

1- .u 4 . 7l Tn,- ut *u theoretical start times. Please see Section I1I.C.4 above, 

hmits that are set on the respective queues. This enables the „ , . , i • • . j i- 

♦ 1 •* CO ♦ * 4 *• ^ t ««t- Real time advantageously is incremented on the time hne 

queue control unit 58 to mitiate congestion control action on , ,7 i . . i j . * 

ABR flows when their oer-VC aueues become excessivelv ^ ""^^^ ^^^^ shaped, time multiplexed output 

ABR tlows wnen theii per queues Decome excessively essentially fill the bandwidth of the output link. As 

long. It also permits the queue control unit 58 to identify . . . u ^ u^ 

a ,u , A' *u ' , ft: i^^^t o« fu^t Will be recalled, the maximum number of resolvable time 

flows that are exceeding theu- traffic contract so that an , . ^ . , . , i ^-i i_ j ^ 

I - A Zr^^ f»^* «o„ K« i^.r^\r^A tr. slots mto wfaich the scheduler 62 can map the head of queue 

appropriate policing function (not shown) can be invoked to , . . « • u f *• r*u 

J 1 11 r u 1- /fi members of the respective flows IS based on the ratio of the 

drop or log cells of such non-comphant flows. . • r * • • 

^ * ^1 ■ 1 jj j» maximum permissible frequency to the minimum permis- 

An admission controUer 61 monitors the "added mes- 35 ^-^^^^ frequency of those flows. Thus, the rate at which real 

sages that are returned by the queue control umt 58 to cause ^-^^ ^ incremented from bucket-to-bucket is a rationale 

a scheduler 62 to schedule the head of queue ceUs for the multiple of the cell rate 

non-empty per-VC queues on a calendar queue 63 for Data transport units residing in time slots which represent 

transmission at scheduled times. Suitably, the scheduler 62 ^-^^^ ^^^^ ^^^j-^^. ^^^^ ^^^^j ^^^^^^ ^^^j ^^^^ 

employs a per-VC virtual clock to schedule these head of ^ference for the time line 66 are eligible for transmission 

queue cells on the calendar queue 63 in accordance with ^^^^ therefore, are linked into a transport list 65 as previ- 

respective virtual finishmg times, VT(f, j+1), that the sched- ^^jy described. However, those data transport units which 

uler 62 computes for them (or, alternatively, "virtual start ^^^-^^ ^^^^ ^^^^ jj^^j associated with later time slots 

times*'). Please see Section III.C.l above. jj^^ ^^^^^^ ^ pending state until system real 

The calendar queue 63 tracks system "real time" or 45 time advances sufficiently to reach those time slots. To avoid 

"current time" to prevent any of the scheduled cells from rollover ambiguities, the time line 66 is designed to ensure 

being released for transmission prior to its scheduled time. that all references to earlier scheduled data transport limits 

In other words, the scheduler 62 and the calendar queue 63 are removed from each time slot before any references to 

implement a stalled virtual clock so that the cells that are later scheduled transport units are inserted therein in antici- 

scheduled for transmission are released for transmission 50 pation of the next scan. 

only when system real time has at least reached their While the above-described arrangement effectively 

respective scheduled transmission times. As illustrated, con- shapes conforming^ DBR/CBR ATM flows to the PCRs 

nections having cells that have been released for transmis- specified by their traffic contracts, it does not aid in bringing 

sion by the calendar queue 63 are linked into a link list of the Cell Delay Variation (CDVs) of those flows into con- 

connections that have cells ready for transmission on a 55 formity with the Xp^R parameters of their traffic contracts, 

transmit list 65. C. Multiple Priority Levels for Minimizing Relative CDV 

The calendar queue 63 notifies the flow control unit 55 In accordance with the present invention, data transport 

whenever it releases a ceU for transmission on any given units that are delivered to a multiplexing point, such as an 

connection. The flow control unit 55, in turn, requests the output port of an ATM switch, by flows having different 

reference to the next cefl (i.e., the new head of queue ceU), 60 frequencies are prioritized so that the data transport units of 

if any, on the per-VC queue for the given connection and the higher frequency flows are given transmit priority over 

notifies the admission controller 61 that it should admit this any data transport units of lower frequency flows with which 

reference to the scheduler 62 for scheduling. Thus, the they happen to collide. As shown in FIG. 3, this transmit 

admission controller 61 effectively engages in closed loop priority can be implemented by steering the data transport 

communications with the calendar queue 63 to ensure that 65 units that are admitted for scheduling by a stalled virtual 

the head of queue cells which it admits for scheduling clock scheduling mechanism 63 or the like to one or another 

thereon are admitted to the exclusion of all other cells in the of a pluraHty of priority rank ordered time-lines 66a-66e or 
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output FIFO queues based on the frequencies of the flows to 
which those respective data transport units belong. For 
example, for an ATM switch, it may be advisable to imple- 
ment on the order of five different frequency dependent/class 
of service dependent output priorities, including (1) a top 5 
priority for cells from flows that have negotiated output rates 
of at least Vie of the full rate of the output link (i.e., its 
^S&tgalc bandwidth), (2) a second priority for cells form 
flows having negotiated output rates ranging from Vie to Vise 
of the output link rate, and (3) a third priority for cells from lO 
flows having negotiated output rates ranging from V^se to 
V4096 of the link rate. The lower two priorities then suitably 
are established for ABR connections that have non-zero 
negotiated MCR rates and for UBR connections and ABR 
connections that have MCR rates of 0, respectively. 15 

As will be appreciated the present invention effectively 
reduces the CDVs of the higher frequency flows, without 
materially increasing the CDVs of the lower frequency 
flows. As a general rule, the CDV that is tolerable is a 
function of the negotiated rate for a flow. For example, a 20 
CDV of 100 cell times is very large with respect to an 
expected emission interval of one cell every 10 cells, but 
generally insignificant if the negotiated emission interval is 
only one ceU every 2029 cells. 

When a calendar queue mechanism is employed to sched- 25 
ule the data transport units or cells of the different frequency 
flows for transmission, the high frequency high priority 
flows need to be resolved to the precision of a single cell 
time while being scheduled to achieve an acceptably low 
CDV, but the low frequency/low priority flows can be more 30 
coarsely resolved to a precision of, say, 16 cell times. This 
means that the number of time slots on the calendar queue 
63 can be reduced. This enables the amount of memory that 
is required to implement the calendar queue 63 to be reduced 
at the cost of losing some typically unneeded precision in the 35 
scheduling of the head of queue cells of the lower firequency 
flows. 

It is to be understood that the frequency based prioritiza- 
tion technique which the present invention provide for 
resolving transmission conflicts at multiplex points among 40 
flows of different nominally fixed frequencies may be 
employed in many different .applications for reducing the 
relative jitter of the flows, including in applications having 
work conserving per- flow output queues for feeding cells or 
other data transport units into such a multiplex point. 45 
D. Traffic Shaping to SCR/IBT and PCR Parameters for Real 
Time and Non-Real Time SBRA^^R Service 

As previously pointed out, an SBR or rt — SBR flow 
(equivalent to a VBR flow) conforms with its traffic contract 
if it not only conforms to its negotiated GCRA (1/PCR, 50 
'^pcr) but also to its negotiated GCRA (1/SCR, Xj^j). In 
accordance with this invention, it has been recognized that 
a stalled virtual clock calendar queue 63 (FIG. 3) can be 
employed to bring one or more flows into conformance with 
such a traffic contract at the egress of a network element, 55 
such as an output port of the ATM switch 21 (FIG. 1). To that 
end, the flow of cells (i.e., data transport units) from the 
output of the calendar queue are shaped to the negotiated 
PCR parameter for the flow unless and until GCRA (1/SCR, 
Xfsj) is true, at which point the output flow is shaped to its 60 
negotiated SCR parameter. 

As shown in FIG. 4, one way to accomplish this is to 
enqueue admitting successive cells of a flow, f, into the 
associated per-VC queue for scheduling by the scheduler 62 
at an initial rate determined by the negotiated PCR param- 65 
eter for that flow. If it is determined at 75 that the cells of the 
flow f are being output at a rate conforming to the GCRA 



(1/SCR, XfQj) for that flow, the scheduler 62 is controlled to 
continue to schedule subsequent ceUs of the flow, f, on the 
calendar queue 63 in accordance with its PCR parameter. On 
the other hand, if it is determined at 75 that the cells of the 
flow, f, are being output from the calendar queue 63 at a rate 
that does not conform to the GCRA (1/SCR, x,sr) for the 
flow, the scheduler 62 then is controlled to schedule these 
subsequent cells on the calendar queue 63 in accordance 
with the negotiated SCR parameter for the flow, f. As will be 
appreciated, the incremental amount by which the virtual 
clock for each successive ceU of the flow f is incremented in 
the connection record for the flow f is easy to switch back 
and forth as required between the PCR emission interval 
T«1/PCR and the SCR emission interval T=1/SCR. 

Alternatively, as shown in FIG. 5, the admission control- 
ler may admit cells firom the flow f for scheduling by the 
scheduler 62 on a relatively low priority stalled virtual clock 
calendar queue 81 or on a relatively high priority stalled 
virtual clock calendar queue 82. If it is determined at 75 that 
the aggregate rate at which the cells of this flow f are being 
output form these calendar queues 81 and 82, conforms to 
GCRA (1/SCR, X/jjt), incoming cells of the flow f then are 
scheduled on the low priority calendar queue 81 by the 
scheduler 62 in accordance with the negotiated PCR for the 
flow. On the other hand, if it is determined at 75 that the rate 
at which the cells of the flow f are being output does not 
conform to GCRA (1/SCR, Zj^j), incoming cells of that flow 
then are scheduled on the high priority calendar queue 82 by 
the scheduler 62 in accordance with the negotiated SCR 
parameter. As shown, steering logic 83 steers the output of 
the scheduler 62 to the high priority or low priority calendar 
queues 81 or 82 depending on whether the GCRA (1/SCR, 
Tj.jyj.) test 75 returns a true or false state. 

Any transmission collisions that occur are resolved in 
favor of outputting the SCR scheduled cells from the high 
priority queue 82 prior to the PCR scheduled cells from the 
low priority queue 82. This enables earUer scheduled cells to 
be output in advance of subsequently scheduled cells, 
thereby preserving cell order. 

E. Traffic Shaping to MCR and PCR Parameters for ABR 
Service in Networks Having Source-to-Destination ABR 
Control Loops 

As will be recalled, the standard ABR traffic contract 
contemplates explicitly or implicitly specified PCR and 
MCR parameters, where MCR may equal 0. An ABR flow 
that conforms to its GCRA (1/MCR, t^cr) is valid and 
entiUed to service, but the network may not be giving the 
flow the bandwidth to which it is entitled. Contrastingly, a 
flow which violates its GCRA (1/PCR, x^^^j^ is non- 
conforming. Thus, ABR connections or flows that are routed 
to the output of a network or network element, such as by the 
Al'M switch 21, advantageously are shaped to ensure: (1) 
that the ABR flows with non-zero MCR guarantees receive 
adequate output bandwidth to effectively satisfy those guar- 
antees; and (2) that none of the ABR flows violate their PCR 
commitments at such output. In networks that have multiple 
multiplex points, it is useful to perform this shaping at each 
of the multiplex points. 

FIG, 6 illustrates one technique for shaping ABR flows or 
VC connections to conform them to their traffic contracts at 
an output of a network or network element. As shown, the 
rates at which the respective flows are requesting output 
service are monitored; first to determine at 85 whether the 
flows are or are not GCRA (1/MCR, x^^j^) compliant, and 
then to determine at 86 whether the GCRA (1/MCR, Xj^cr) 
non-compliant flows are or are not GCRA (1/PCR, Xj^^^) 
compliant. Control signals reflecting the results of these 
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cascaded tests 85 and 86 are fed back to the scheduler 62 and At enqueue: 

its associated steering logic 87 to permit appropriate rate enqueue in high priority queue with rate MCR; 

shaping adjustments to be made to the rates at which the data if GCRA (1/PCR, Xp^id compliant then enqueue in 

transport units (e.g., cells) of the respective flows are fed to low-priority work conserving queue; 

the output. As will be appreciated, the desired state is for all 5 else enqueue in low-priority non-work conserving 

of the ABR flows to be GCRA (1/MCR, x^^id non- q^eue based on PGR emission interval; 

compliant (or just barely compliant) and GCRA (1/PCR, dequeue* 

false/compliant because those algorithms test for dequeuing from high priority queue then delink from 

mimmum and maximum acceptable now rates, respectively. , . ^ , ^ , r ... • • 

More particularly, GCRA (1/MCR, x„^^ compliant P"°"'y ^"^"^ ^^^"^ ^""^ P"°"'y 
flows may not be receiving the output bandwidth that they - o^.?,^^* -n t . • 
need to satisfy their MCR guarantees. Therefore, when such ^ lU^strates still another altemative for shaping ABR 
a flow is identified at 85, the scheduler 62 and its steering connections or flows. In this embodiment, the output band- 
logic 87 are set-up to schedule subsequently received ref- ^^^^ that is provided for each MCR non-compliant ABR 
erences to data transport units (e.g., cells) of that flow on a ^^"^ that has a non-zero MCR guarantee is dynamically 
relatively high priority stalled virtual clock calendar queue adjusted based on the average spacing between ABR cells on 
88 for output at a rate somewhat in excess of its guaranteed the output link (i.e., at the transmit side of a line interface) 
MCR. For example, these subsequently received cell or and the current length of the ABR "ready queue." The 
packet references suitably are scheduled on the calendar current length, Len, of this ABR "ready queue" conveniently 
queue 88 for emission at a rate that is determined by is determined by measuring the length of the work conserv- 
multiplying the MCR for the flow by a predetermined 20 ing queue on the ABR transmit lists 65^/ and 65e as at 93. On 
speed-up factor. This causes the network to give these the other hand, the average spacing, S, between ABR cells 
additional potentially "lagging" flows the additional output on the calendar queue can be determined at 99 dividing the 
bandwidth that is needed to satisfy their MCR guarantees. bandwidth that is available for ABR flows after allowance is 

GCRA (1/MCR, Xj^cid non-compUant ABR flows are uj^de for any non-ABR flows. The bandwidth required by 

further tested at 86 to determine whether they are denianding 25 these non-ABR flows suitably is determined by computing 

fr^Z ^^fP^^^^y unacceptably high at 94 the aggregate bandwidth needed to service the transmit 

GCRA (l^CR, v^^) comphant or non-compliant rates, ^.^^ ^5 ^^^^^^ non-ABR flows, 

respectively. Tl^e results of this test are captured by another ^^^^1^^ ^ determined at 94 that 50% of the 

control signal which is fed back to the scheduler 62 and its t^^ndwidth of the output link is available for the ABR flows, 

steermg logic 87 to set them up for rate sensitive enqueuing 30 fo^^^^ ^^^^ j^e average space, S, between ABR cells or 

?f .^^^.,'''^'^'^1'°^^^ '"^^'"^^ references to these GCRA ^^^^ ^ units on the output link is two ceU times. This 

(1/MCR T^^^) non-comphant flows on a lower priority information, together with the computed length, Len, of the 

queue 89 Specifically, the suteequen^^^^^ . ^^^^^^^ ^^^^^ j^CR 

to a flow that is found to be GCR A (1/PCR, ^/"Pl^^nt non-compliant ABR flows to be optimized by having the 

at 86 are appended to the tail of the transmit lis^ 35 ^^^^^^^^^ ^^^^^^^ ^^^^^ references for those 

flows with MCR guarantees for it is serviced m round-robin g^^^ non-work conserving calendar queue 89 at 

^""^ o/T"* ^°u^' respective emissions intervals T, of: 
(1/PCR, Xpcid non-comphant at 86, the subsequently 

received references for that flow are scheduled by the T,=Min (s„ Max (p„ r)) (3) 
scheduler 62 on a non-work conserving, stalled virtual clock 

calendar queue in accordance with the PCR for that flow. Where: R-an estimated round robin ABR 

thereby imposing a PCR Umit on the flow. service time (Recurrent time+S*Len); 

More generally, it will be evident that the embodiment of p^^t^e earliest service contract compliant departure for 

FIG. 6 implements the foUowmg ABR traffic shaping algo- the next ceU of flow t (P,=current time+l/PCR); and 

r^^^™- 45 S,==the latest service contract compliant departure for 

If GCRA (1/MCR, x^cid compliant then enqueue in high the next cell of flow t (S,=current time+l/MCR). 

priority queue at rate MCR*speedup p. Traffic Shaping for Segmented ABR Connections 

else if GCRA (1/PCR, XpciO compliant then enqueue in As is known, ABR connections may operate with source- 
low-priority work conserving queue to-destination control loops, such as shown in FIG. 9 at 101, 

else enqueue in low-priority non-work conserving queue 50 or with segmented control loops, such as shown in FIG. 10 

based on PCR emission interval. at 102. When segmented control loops are employed, each 

However, alternatives that are based on similar and dif- of the separately controlled segments functions as a virtual 

ferent principles will suggest themselves. source, VS, for the next segment. Thus, each ABR control 

For example, as shown in FIG. 7, one potentially attrac- segment (except the first) is sourced by a virtual source 

tive alternative is to enqueue the references to the data 55 which assumes the behavior of an ABR source endpoint. 

transport units or cells of flows that are found to be GCRA Backward resource management (RM) cells are removed 

(1/MCR, Tjv^cr) non-compliant and GCRA (1/PCR, Xpc/O from the control loop when they are received by a virtual 

compliant at 85 and 86 (FIG, 4), respectively, in both (1) the source. However, each source or virtual source can compute 

high priority calendar queue 88 at a MCR scheduling an estimate of the available network bandwidth to its des- 

interval, and (2) the work conserving region of the lower 60 tination or virtual destination, VD, based on this feedback 

priority queue. This creates a "race" condition because which, in turn, can be expressed as an Allowed Cell Rate 

whenever one instance of such a dual queued reference is (ACR) on a per-segment basis. Therefore, ABR traffic 

reached for service in either of the queues, a delinker 90 shaping advantageously is carried out segment-by-segment 

removes the other instance of that reference from the other for these ABR connections with segmented control loops, 

queue, 'ITiis means that a tighter bound may be maintained 65 using a maximum allowable per-flow data transport unit or 

on the tolerance, x. 'Ilie traffic shaping algorithm of this cell rate of Max(MCR, Min(ACR, PCR), llie embodiments 

modified embodiment is: of FIGS. 6-8 can implement such traffic shaping by setting 
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PCR' equal to Max(MCR, Mm(ACR, PGR) and then by 
using PGR' in place of PGR in the coniputations. 

G. Traffic Shaping to PGR for UBR Service 

The embodinaent of FIG. 6 can be simplified to shape 
traffic to a specified PGR for UBR service. UBR service 
contemplates an explicitly or implicitly specified PGR, but 
no MGR. Thus, the provision made to enforce the ABR 
MGR guarantees may be eliminated, which means that the 
incoming references to the data transport units or cells of a 
UBR flow or connection may be enqueued on the queue 89 
in accordance with the flowing enqueuing rules: 

if GGRA(1/PCR, Xp^f^) compliant then enqueue in bucket 
at end of work conserving region 
else enqueue in non-work conserving region based on 
PGR emission interval. 

H. Rate Shaping for Servicing Multiple Physical Layers 
From a Single Queuing Point 

As is known, Utopia 2 is a standard interface for con- 
necting multiple physical layers, such as at 111-116 in FIG. 
11, to a single ATM output queuing point. Typically, each of 
these physical layers 111-116 run at the same rate, but this 
run rate is only a fraction of the cell rate at the output of the 
queuing point (e.g., one of the output ports of the ATM 
switch 21). Thus, it may be necessary to accommodate such 
a rate mismatch in order to properly shape the output traffic 
to the traffic contracts governing the transfer of cells to the 
respective physical layers 111-116. 

It has been found that the above-described stalled virtual 
clock techniques can be adopted to shape the traffic at the 
output of a relatively high data rate queuing point to the 
traffic contract requirements of a plurality of slower data rate 
physical layers 111-116, without any extra buffering of the 
traffic at the point of the rate mismatch. To accomplish this, 
the scheduler 62 schedules the data transport units for the 
flows (e.g., connections) to the different physical layers on 
respective actual or virtual per-physical layer stalled virtual 
clock calendar queues. The time slot-to -time slot granularity 
of these calendar queues suitably is selected to equal the 
ratio of MP to the data rate (i.e., cell rate) at the output of the 
queuing point, where MP is the lowest integer (conveniently 
a power of two) that is equal to or greater than the number 
of physical layers 111-116. This increased granularity effec- 
tively steps down the rate at which real time advances on the 
calendar queues to a rate which at least approximates the 
data rate of the physical layers 111-116. As will be 
appreciated, the time slot-to-time slot granularity of the per 
physical layer calendar queues may be further increased as 
described in Section VI.C above if the reduced precision in 
the scheduling of the lower frequency flows is tolerable. 

As shown in FIG. 12, the per physical layer calendar 
queues of any given priority may be interleaved on a single 
time line 109. Thus, when a connection to a physical layer 
transmits a cell and reschedules the next cell for one cell 
time later than current time, this next cell is scheduled MP 
cell limes after current time on the time line 109 to step the 
effective emission rate of the calendar queue for that par- 
ticular physical layer down to or about its run rate. 

As illustrated,jhc~time ~Unc'1 09~iis'corifigurcd4 
si^T)h55icd:l^lsill-l^^ ihere still is a minor rale 

mismatch""even after the effective emission rates of the 
calendar queues have been stepped down, as taught above, 
to approximate the run rates of the physical devices 111-116. 
Fortunately, however, this minor rate mismatch can be fully 
compensated for by adjusting the per-flow constants for the 
virtual clock computations (see II I.I above), and by the 
Utopia 2 flow control for ABR and UBR connections. 
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What is claimed: 

1. In a packet switched communication system including 
a traffic shaper and a relatively high data rate port that is time 
multiplexed to feed packet traffic to a plurality of relatively 

5 low data rate physical layers, said packet switched commu- 
nication system comprising: 

a queuing mechanism, said queuing mechanism organiz- 
ing packet traffic to said physical layers in respective 
10 queues in accordance with an oldest pending packet at 
head of queue order; 

a non-work conserving calendar queue mechanism 
coupled to said queuing mechanism, said non-work 
conserving calendar queue mechanism controlling the 

■"^ emission of packet traffic to said physical layers; and 
a scheduling mechanism coupled to said queuing mecha- 
nism and to said calendar queue mechanism, said 
scheduling mechanism scheduling packet traffic for 

20 emission to respective physical layers on said non-work 
conserving calendar queue mechanism at a rate sub- 
stantially equal to or less than a data rate of said 
physical layers. 

2. The packet switched communication system of claim 1, 
25 wherein said calendar queue mechanism includes respective 

calendar queues for said physical layers, at least one of said 
calendar queues has a scheduling granularity that is sub- 
slantiaUy equal to or greater than the number of said physical 
layers divided by the data rate of said port. 
^° 3. The packet switched communication system of claim 2, 
wherein said scheduling granularity is substantially equal to 
a lowest power of two integer, 

4. The packet switched communication system of claim 1, 
wherein said calendar queue mechanism is a single calendar 
queue which has a scheduling granularity that is substan- 
tially equal to or greater than one divided by the data rate of 
said port. 

5. The packet switched communication system of any of 
claims 1-4 further including at least one other traffic shaper 
in a path leading to said port for shaping multiple flows of 
packets to respective network traffic contracts. 

6. In a packet switched communication system having a 
port that is time multiplexed to distribute packet traffic 
composed of multiple flows to a plurality of limited band- 
width paths, where at least one of said paths is eligible to 
receive said flows, a traffic shaper comprising: 

a queuing mechanism, said queuing mechanism organiz- 
ing packet traffic on said paths in respective queues in 
50 accordance with an oldest pending packet at head of 
queue order; 

a non-work conserving calendar queue mechanism 
coupled to said queuing mechanism, said non-work 
conserving calendar queue mechanism controlling the 

55 emission of packet traffic on said paths; and 

a scheduling mechanism coupled to said queuing mecha- 
nism and to said calendar queue mechanism, said 
scheduling mechanism scheduling packet traffic for 
emission on said paths on said non-work conserving 

60 calendar queue mechanism at a rate substantially equal 
to or less than a data rate of said paths. 

7. The traffic shaper of claim 6 further including a second 
traffic shaper for shaping multiple flows of packets to 
respective network traffic contracts. 

65 

♦ * * « * 
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